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ABSTRACT 

A  renaissance  in  the  research  and  development  of  computer-based,  tutoring  systems  over  the  last  ten  years  is 
motivating  scientists  to  ponder  the  application  of  intelligent  tutors  and  coaches  in  more  challenging  team  training 
problem  spaces  where  human  tutors  are  either  unavailable  or  impractical.  This  paper  reviews  some  of  the  challenges 
and  emerging  technologies  (tools  and  methods)  that  might  influence  the  development  of  adaptive,  intelligent  tutors 
for  geographically-distributed  team  training.  Team  tutoring  presents  many  challenges.  Even  human  tutors  struggle  to 
develop  team  cohesion,  coordinate  roles  and  responsibilities  of  team  members  and  assess  their  contributions 
(Sottilare,  2010).  Computer-based  tutors  face  additional  challenges:  sensing  and  assessing  the  cognitive  state 
(including  affect)  of  each  team  member  (Sottilare  and  Proctor,  in  press',  D'Mello  and  Graesser,  2007)  in  near  real¬ 
time  to  understand  each  team  member'^  readiness  to  learn  (e.g.,  their  engagement  and  motivation);  measuring  team 
performance;  perceiving  and  weighing  team  member  contributions  to  team  performance;  and  selecting  instructional 
strategies  that  will  optimize  team  performance.  Emerging  sensing  technologies  are  showing  promise  as  enablers  of 
computer-based  perception  of  each  team  member"s  behavior  and  physiology  with  the  goal  of  predicting  unobserved 
variables  (e.g.,  cognitive  state).  Along  with  performance  measures,  historical  and  self-reported  data,  behavioral  and 
physiological  measures  can  provide  the  tutor  with  the  information  needed  to  model  the  trainee"s  state  and  their 
relationship  with  other  team  members  and  the  tutor.  Accurate  (and  timely)  trainee  and  team  state  information  (e.g., 
performance,  competency,  trust)  are  considered  to  be  determining  factors  for  the  team  tutor  to  select  appropriate 
instructional  strategies  (e.g.,  support,  direction)  for  optimal  team  performance.  Design  goals,  ongoing 
experimentation,  and  potential  applications  of  computer-based  team  tutors  are  also  discussed. 
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INTRODUCTION 

The  military  has  exploited  artificial  intelligence  within 
existing  technologies  (e.g.,  virtual  worlds,  games, 
virtual  humans)  to  provide/ supplement  team  training, 
but  limited  attention  has  been  given  to  the  need  for 
computer-based  tutoring  during  distributed  team 
training  exercises  where  human  tutors  are  either 
unavailable  or  impractical  (e.g.,  not  cost-effective). 

In  the  1980"s  and  early  1990"s  there  were  significant 
military  research  investments  in  computer-based 
tutoring  technologies  for  individual  training  in  well- 
defined  domains  (e.g.,  development  of  procedural 
knowledge).  These  investments  cooled  off  as  the 
promise  of  ubiquitous  tutoring  systems  failed  to 
materialize  in  the  military  training  domain  where  the 
emphasis  tends  to  be  focused  on  collective  training. 
Over  the  last  ten  year,  computer-based  tutoring  systems 
have  advanced  along  with  learner  and  expert  modeling, 
authoring  tools  and  methods  to  select  instructional 
strategies.  The  maturity  of  technologies  for  individual 
tutoring  systems  bodes  well  for  some  near-term 
solutions  to  team  tutoring. 

Significant  technical  challenges  remain  in  providing 
geographically-distributed  team  training.  This  paper 
evaluates  the  challenges  related  to  the  management  of 
training  vice  its  development  and  distribution  to 
trainees.  Specifically,  we  address  the  functions  of  an 
intelligent  tutoring  system  (ITS)  for  distributed  team 
training;  how  these  functions  might  differ  from 
distributed  training  for  individuals;  current 
capabilities/methods  to  support  distributed  team 
training;  and  areas  for  future  research. 

Thoughts  on  Team  Training 

What  is  needed  now  is  a  means  to  understand:  the  state 
(e.g.,  bored,  frustrated  or  engaged)  of  each  distributed 
team  member;  their  individual  performance;  the 
interactions  (e.g.,  communication)  of  team  members; 
and  the  contribution  of  individual  state,  individual 
performance,  and  team  member  interactions  to  the 
collective  performance  of  the  team.  Challenges  include 


some  of  those  documented  for  individual  ITS,  but  there 
are  also  some  new  ones  to  be  overcome  if  a 
collaborative  training  environment  is  to  be  fostered. 
Individual  ITS  must  be  capable  of  supporting  the 
training  needs  of  each  team  member,  but  also  be  able  to 
communicate  with  other  individual  ITS  regarding 
progress  toward  team  goals,  individual  contributions 
toward  team  goals  and  the  formulation  of  instructional 
strategies  (e.g.,  support,  directions,  questions,  feedback 
or  hints)  and  interventions  for  the  collective  team 
and/or  individual  members.  A  large  part  of  the  process 
to  determine  an  optimal  instructional  strategy  is  the 
tutor"s  ability  to  sense  behavioral  and  physiological 
cues  and  use  those  cues  to  classify/predict  individual 
trainee  state  (e.g.,  emotions,  beliefs,  desires  or 
intentions).  The  premise  being  that  the  better  the  ITS 
understands  the  trainee  (or  the  more  comprehensive  the 
trainee  model  is),  the  better  the  ITS  will  be  able  to 
select  appropriate  instruction  and  feedback. 

Team  ITS  also  have  the  added  workload  of 
coordinating  the  states  of  individual  team  members  so 
a  more  comprehensive  picture  of  the  team  state  can  be 
developed.  Inputs  to  the  “team  model”  might  include 
the  state  of  trust  between  individual  team  members, 
progress  toward  team  goals,  reassessment  of  team 
goals  based  on  priorities  and  the  distribution  of 
workload. 

Considerations  for  Team  Tutoring 

When  trainees  effectively  learn  in  groups,  they  can 
encourage  each  other  to  ask  questions,  explain/justify 
their  opinions  and  reasoning,  and  actively  reflect  upon 
their  knowledge.  Research  has  shown  such  situations 
to  increase  group  performance  and  individual  learning 
outcomes  (especially  motivation  and  engagement  - 
Tchounikine,  Rummel,  and  McLaren,  2010). 
However,  these  benefits  can  only  be  achieved  in  well¬ 
functioning,  actively  learning  teams  (Jarboe,  1996; 
Soller,  2001).  While  some  teams  may  have  successful 
interaction  and  communication  naturally,  others  may 
be  incapable  of  developing  a  balance  of  participation, 
leadership,  understanding,  and  encouragement  (Soller, 
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2001).  This  dysfunction  can  rapidly  degrade  group  and 
individual  performance,  motivation,  and  engagement. 

In  order  for  a  computer-based  team  intelligent  tutoring 
system  to  be  successful,  three  primary  factors  should 
be  understood  and  addressed  in  its  design:  (a)  the 
accountability  collaborative  learning  interactions  and 
communications;  (b)  the  complexity  of  training  tasks; 
and  (c)  the  physical  distribution  of  the  team. 

Foremost  of  the  three  factors  with  respect  to 
establishing  effective  collaborative  learning  is  the 
uncertainty  and  dynamic  nature  of  team  interaction  and 
communication.  New  team  members  may  enter  the 
team  and  old  ones  may  leave.  The  social  interaction 
among  team  members  that  is  necessary  for  trust¬ 
building  will  not  always  foster  learning  (Brown  and 
Palincsar,  1989).  Traditionally,  trainees  view  learning 
as  an  independent  and  mildly  competitive  activity. 
Trainees  typically  do  not  ask  for  help  from  their  peers 
for  fear  of  appearing  incompetent  or  dependent. 
Furthermore,  peers  tend  to  work  together  with  the  aim 
of  just  accomplishing  tasks  (e.g.,  finding  the  right 
answers)  instead  of  facilitating  each  other"s  learning. 
A  team'k  learning  potential  is  maximized  when  each 
individual  actively  participates  in  the  learning  task, 
thereby  increasing  the  probability  all  trainees 
understand  the  learning  material  and  no  one  is  left 
behind.  To  promote  active  participation,  a  successful 
team  ITS  must  be  able  to  (a)  encourage  the  individual 
trainee  to  exchange  ideas,  information,  and 
perspectives  for  interaction,  and  (b)  provide  real-time 
monitoring  of  individual  and  team  participation  level 
(e.g.,  interaction  analysis),  and  react  to  low 
participation  levels. 

The  second  concern  of  team  tutoring  relates  to  the 
interaction  differences  within  different  learning  tasks. 
Peer  interactions  have  been  found  to  vary  enormously 
in  collaborative  learning  initiatives  within  the  same 
domain  (Brown  and  Palincsar,  1989),  but  the  same 
team  will  certainly  communicate  differently  and 
unpredictably  between  two  different  domains.  One 
aspect  attributing  to  the  variation  in  communication  is 
the  roles  trainees  assign  themselves  within  a  team  to 
accomplish  a  targeted  objective  and  how  they  might 
productively  switch  roles  between  tasks  (Burton, 
1998).  Role  identification  and  switching  is  good  for 
social  grounding  development  and  can  create  an 
environment  for  collaborative  learning  and  more 
effective  communication  (Soller,  Linton,  Goodman  and 
Gaimari,  1998).  However,  not  all  individuals  will  be 
able  to  identify  their  roles  within  the  team  context. 
This  can  often  lead  the  team  falling  off-track  or 
misguiding  each  other.  Even  when  team  members  are 
not  strangers,  they  still  need  support  and  guidance  on 


how  to  work  together.  Furthermore,  as  the 
task/objective  becomes  more  complex,  effective 
communication  within  the  team  becomes  more 
important  and  more  complex.  To  address  this  second 
concern,  a  team  ITS  should  diagnose  and  redirect 
incorrect  solution  paths,  divide  complex  tasks  into  sub¬ 
tasks  for  which  each  person  could  be  responsible  for  a 
sub-task  or  the  entire  group  tackles  each  sub-task 
together,  and  instruct/guide  the  team  in  the  beginning 
on  how  collaboration  should  occur  (Wu,  Farrell  and 
Singley,  2002). 

Team  locality  is  the  third  primary  concern  of  team 
tutoring.  Face-to-face  or  local  teams  have  been  found 
to  have  more  favorable  learning  results  over  distributed 
or  virtual  teams  (Andres,  2002;  Warkentin,  Sayeed  and 
Hightower,  1997).  Virtual  teams  exchange  information 
less  effectively  than  face-to-face  groups  (Warkentin, 
Sayeed  and  Hightower,  1997);  however  given 
sufficient  time  to  develop  strong  group  relationship  and 
become  comfortable  with  the  communication 
environment,  virtual  teams  may  communicate  as 
effectively  as  a  face-to-face  team  (Chidambaram, 
1996).  In  military  team  training  and  simulations,  it  may 
be  impractical  for  the  team  to  be  in  a  centralized 
location.  A  mediation  for  the  concern  of  geographical 
distributed  team  tutoring  could  be  for  the  team  ITS  to 
provide  team-building  and  team-readiness  activities 
prior  to  beginning  instruction. 

DESIGNING  A  DISTRIBUTED  TEAM  TUTOR 

A  goal  of  this  research  is  to  develop  a  team  tutor  that 
will  ultimately  eliminate  the  need  for  human  tutors  for 
distributed  training  where  interaction  and  direct  access 
to  trainees  is  limited.  A  large  part  of  developing  a 
distributed  team  tutor  is  defining  which  functions  are 
desirable  within  individual  ITS.  We  began  with  a 
model  of  an  individual  tutor  and  then  explored  which 
interactions  between  individual  tutors  were  needed  and 
which  new  processes  were  needed  to  realize  a 
distributed  team  tutor. 

Building  upon  an  Individual  ITS  Model 

As  a  starting  point  for  our  team  tutor  model,  we 
adopted  an  individual  ITS  model  based  in  part  on 
Beck,  Stern  and  Haugsjaa"s  (1996)  Intelligent  Tutoring 
System  Model  and  in  part  on  procedural  reasoning 
systems  (Georgeff  and  Lansky,  1987;  Parunak,  Bisson, 
Brueckner,  Matthews  and  Sauter,  2006)  which  were 
used  to  model  affect  (e.g.,  emotions)  within  virtual 
characters.  Sottilare  (2010)  adapted  Beck,  Stern  and 
Haugsjaa's  (1996)  ITS  model  resulting  in  the 
individual  affective  ITS  Model.  This  model  was  a  basic 
building  block  for  our  initial  team  tutoring  model  due 
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to  its  inclusion  of  affect,  its  extensibility,  and  its 
emphasis  on  a  more  comprehensive  trainee  model.  In 
2011,  additional  emphasis  on  learning  factors  (e.g., 
remembering,  understanding,  analyzing)  and 


integration  with  a  more  long-term  trainee  model 
resulted  in  the  enhanced  tutoring  model  shown  in 
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Figure  1:  Enhanced  Tutoring  Model  in  the  Generalized  Intelligent  Framework  for  Tutoring  (GIFT) 


Figure  1.  This  figure  illustrates  the  Generalized 
Intelligent  Framework  for  Tutors  (GIFT)  that  is  being 
developed  by  the  Army  Research  Laboratory  to  test 
tutoring  methodologies  and  their  influence  on  learning. 

The  enhanced  trainee  model  within  GIFT  accounts  for 
competence,  affect,  physiological  and  behavioral  data 
to  assess  new  emotional  states.  Progress  toward 
training  objectives  (e.g.,  knowledge  or  skill 
acquisition)  is  used  to  assess  changes  in  competence 
level  while  competence  level  and  new  emotional  states 
are  used  to  make  decisions  about  instructional 
strategies. 

Team  Tutoring  Design  Goals 


Five  goals  were  established  for  the  design  of  our  team 
tutor.  The  team  tutor  should  be: 

•  accurate :  methods  used  to  assess  affect,  trust, 
etc.  should  correctly  assess  trainee  state  and 
produce  a  minimum  of  false  negatives 

•  low-cost,  the  tutor  should  be  software 
intensive  and  require  minimum  additional 
hardware  beyond  the  training  platform  (e.g., 
laptop  or  other  mobile  computing  device) 

•  portable',  the  tutor  should  be  able  to  be  hosted 
on  a  standard  laptop  and  eventually  other 
mobile  computing  devices  (e.g.  IPhone, 
Android)  to  provide  a  mobile  team  training 
capability 

•  real-time',  interaction  with  the  distributed 
tutor"s  architecture  should  be  in  near  real-time 
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Figure  2.  Enhanced  Team  Tutoring  Model  in  the  GIFT 
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•  so  as  not  to  adversely  influence  individual  or 
team  learning  or  performance 

•  unobtrusive :  methods  should  be  passive  in 
that  they  minimize  disruption  of  the  training 
process  and  be  compatible  with  available 
technology  (e.g.,  laptops  or  other  mobile 
computing  platforms) 

Other  considerations  for  designing  and  developing  an 
effective  team  tutor  are  discussed  below. 

Interactions  and  State  Models  for  Team  Tutors 

This  section  discusses  which  information  should  be 
shared  between  individual  computer-based  tutors  and 
considers  which  state  models  should  be  maintained 
within  the  proposed  team  tutoring  model.  In  a 
distributed  simulation  framework  (e.g.,  High  Level 
Architecture  or  Distributed  Interactive  Simulation) 
each  local  training  simulation  shares  interaction  data 
with  other  simulations  to  support  a  common  view  of 
the  world  or  shared  synthetic  environment.  We  have 
adopted  a  similar  framework  to  support  our  team  tutor 
model.  Consideration  should  be  made  to  exchange  any 
data  that  might  be  relevant  to  team  performance 
including  trainee  communications,  trainee  behaviors 
(e.g.,  actions,  performance),  trainee  affect,  and  any 
changes  in  state  models.  The  type  of  information  to  be 


exchanged  and  frequency/triggers  for  exchanges  (e.g., 
periodically  or  based  on  events/changes)  are  discussed 
below.  The  notional  team  tutoring  model  in  Fig.  2 
builds  on  Sottilare"s  (2010)  team  state  model  and 
illustrates  interactions  between  geographically 
distributed  autonomous  tutoring  systems  to  maintain  a 
common  picture  of  team  performance  and  learning. 

Team  Performance  State  Models 

Team  Performance  updates  are  event  driven  based  on 
changes  to  the  Team  Performance  State  Model.  As 
team  members  complete  assigned  tasks,  progress 
toward  team  goals  is  registered  within  the  individual 
ITS"  Team  Performance  State  Model  and  the  ITS  then 
generates  an  update  message  to  the  other  ITS  so  all 
Team  Performance  State  Models  are  synchronized. 

Team  Competency  State  Models 

This  model  provides  an  index  of  team  competency 
based  on  a  composite  of  the  competence  levels  of 
individual  team  members.  Successful/unsuccessful 
performance  influences  individual  competence  and 
may  influence  team  competence.  Any  significant 
changes  in  individual  performance  of  team  tasks  are 
assessed  by  the  individual  competency  state  model  to 
determine  if  the  threshold  has  been  met  to  change 
individual  competency  (e.g.,  beginner,  journeyman  or 
expert).  Changes  in  individual  competency  may  or  may 
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not  be  of  sufficient  significance  to  affect  a  change  in 
the  team  competency  state  model,  but  if  a  change  in  the 
team  competency  state  model  occurs,  a  message  is 
generated  to  update  the  team  competency  state  models 
of  the  other  team  members. 

Team  Cognitive  State  Models 

This  state  model  is  a  compound  model  of  the  Cognitive 
State  of  all  team  members.  Cognitive  State  models 
already  exist  as  part  of  the  individual  ITS,  but 
synchronization  of  this  information  with  all  the  team 
member"s  ITS  is  critical  in  assessing  the  function  of  the 
team.  Depending  on  the  collaborative  task  and  the  roles 
of  team  members  in  accomplishing  that  task,  the 
cognitive  model  may  be  key  in  determining 
instructional  strategies.  For  example,  for  some  tasks 
the  weakest  understanding  of  the  task  among  team 
members  may  indicate  the  risk  of  completing  the  task 
successfully.  For  other  tasks,  only  key  team  members 
may  need  to  have  higher  understanding  of  the  task  to 
reach  a  successful  outcome. 

Team  Affective  State  Models 

This  state  model  is  a  compound  model  of  the  Affective 
State  of  all  team  members.  Affective  State  models 
already  exist  as  part  of  the  individual  ITS,  but 
distribution  of  this  information  to  all  the  team 
member"s  ITS  is  critical  in  assessing  the  function  of  the 
team.  For  example,  if  team  performance  is  below 
expectations  and  the  affective  state  of  one  or  more 
team  members  is  negative,  knowledge  of  their  state  by 
other  individual  ITS  provides  the  opportunity  to 
prompt  their  associated  team  members  to  take  action 
(e.g.,  communicate  -  support  or  direct). 

Team  Trust  State  Models 

This  team  state  model  is  a  compound  model  of  the  trust 
states  existing  between  team  members.  The  trust 
relationships  are  bi-direction  in  that  Team  Member  ,A" 
may  trust  Team  Member  ,3"  more,  the  same  or  less 
than  Team  Member  „B"  trusts  Team  Member  ,A"  Trust 
is  influenced  by  several  factors  including  perceived 
competency,  perceived  integrity,  perceived 
benevolence,  knowledge  of  the  other  team  members 
(Hung,  Dennis  and  Robert,  2004)  and  perceived 
benefits  of  the  relationship  (Gujral,  DeAngelis,  Fullam 
and  Barber,  2006).  Since  teams  work  toward  common 
goals  where  roles  and  responsibilities  are  distributed, 
perceived  competency  is  an  essential  element  of  team 
performance.  The  perception  that  other  team  members 
may  be  unable  to  perform  their  tasks  is  detrimental  to 
trust  and  team  performance.  Personality  may  also  play 
a  part  in  trust. 

Individuals  with  low  openness  and/or  high  neuroticism 
scores  in  the  Five  Factor  Model  of  Personality 


(McCrae  and  Costa,  1994)  may  have  developed  habits 
unfavorable  to  the  development  of  trust.  Low  openness 
scores  might  indicate  an  unwillingness  to  disclose 
information  while  high  neuroticism  scores  might  result 
in  more  frequent  perception  of  events/interactions  as 
negative.  Positive  or  negative  emotions  can  also 
influence  the  assimilation  of  information  (Linnenbrink 
and  Pintrich,  2002)  and  thereby  communications, 
understanding  and  trust. 

Team  Communication  State  Models 

This  model  is  composed  of  interaction  data  between 
team  members  for  the  purpose  of  observing  team 
cohesion  and  task  execution.  Providing  accurate 
information  in  accordance  with  operating  procedures, 
providing  communications  when  asked,  repeating 
communications  to  ensure  delivery,  sharing 
information  and  acknowledging  receipt  of  information 
are  all  vital  actions  observed  in  teams  with  effective 
communication  skills  (U.S.  Coast  Guard,  1998).  In 
team  settings  communication  among  members  builds 
holistic  situational  awareness  and  coordinates  future 
actions  to  be  carried  out.  Based  on  events  and 
interactions  in  a  scenario,  team  members  are 
responsible  for  updating  one  another  in  real-time. 

In  computer-based  team  training,  tracking 
communication  between  members  can  be  valuable  for 
identifying  causes  in  performance  deficiencies  and 
providing  feedback  in  after-action  reviews  (AAR).  As 
well,  modeling  expert  communication  tactics  for  case- 
based  scenarios  can  mark  distinct  environmental 
factors  that  warrant  team-wide  communication. 
Determining  events  that  should  trigger  communication 
can  allow  for  real-time  interventions  that  emphasize  the 
cause  and  need  for  conveying  information  on  a  team¬ 
wide  level.  Assessing  communication  effectiveness 
among  teams  can  also  inform  the  other  team  state 
models.  Errors  in  statements  and  low  communication 
rates  can  be  used  to  inform  both  the  trust  and 
competency  state  models. 

Challenges  in  the  Development  of  a  Team  Tutor 

In  considering  the  design  of  computer-based  team 
tutors,  we  identified  five  primary  challenges  in 
developing  a  computer-based  team  tutor. 

Challenge  1:  Low  cost,  passive  sensing  of  behavioral 
and  physiological  data 

Just  as  in  individual  tutors,  the  ability  to  accurately 
sense  the  behaviors  and  physiological  responses  of 
trainees  is  a  key  to  building  a  comprehensive  trainee 
model.  This  challenge  is  deepened  when  attempting  to 
connect  specific  behaviors  and  physiological  responses 
to  a  specific  individual  in  another  location  to  determine 
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the  relationship  of  those  team  members  and  whether 
the  tutor  should  intervene  in  some  way. 

Challenge  2:  Classification  of  Affect  and  Trust 

The  ability  to  accurately  classify  the  affective  state  of 
each  trainee  and  their  trust  relationships  will  determine 
the  ability  of  the  tutor  to  select  instruction  and 
feedback  that  is  tailored  to  each  trainee  and  appropriate 
for  the  team. 

Challenge  3:  Selection  of  Instructional  Strategies 

A  significant  challenge  is  the  selection  of  effective 
instructional  strategies  based  on  the  individual  trainee"s 
state  (e.g.,  competence  level  and  affective  state)  that 
will  support  the  trainee"s  individual  needs  and  the 
team's  needs.  At  times,  individual  and  team  needs  may 
conflict.  For  example,  if  speed  is  a  critical  team 
performance  measure  and  an  individual  tutor  finds  it 
necessary  to  provide  remediation  for  a  confused  or 
frustrated  trainee,  what  should  take  priority?  Is  the 
priority  the  needs  of  the  trainee  or  the  team?  If  the 
overall  goal  of  the  training  is  for  team  to  improve,  it 
may  be  important  to  address  individual  team  member 
needs  and  then  move  on  to  team  goals,  but  what  if 
there  are  persistent  issues  with  team  members?  Should 
they  be  addressed  by  the  other  team  members,  a  team 
leader  or  the  team  tutor? 

Challenge  4:  Tracking  Multi-Dimensional  States 

The  size  of  the  team  can  quickly  affect  the  number  of 
states  the  team  tutor  must  synchronize.  For  example, 
the  team  trust  state  model  is  composed  of  the 
relationships  between  each  pair  of  individual  team 
members.  Equation  (1)  shows  the  relationship  between 
the  number  of  team  members,  n,  and  the  number  of 
trust  relationships,  T. 

T  =  n2  -  n  (1) 

The  number  of  trust  relationships  which  must  be 
updated  and  synchronized  for  a  squad  of  nine  Soldiers 
is  72.  Each  of  these  relationships  must  be  derived  from 
the  history  the  two  individuals  (e.g.,  just  met  or  friends 
for  life),  the  frequency  of  communications,  their 
willingness  to  share  information  and  their  affective 
states  (e.g.,  emotions  and  personality).  The  accuracy  of 
these  predictions  should  influence  decisions  by  the 
tutor  during  training  and  inaccurate  modeling  of  trust 
would  likely  be  more  detrimental  than  not  modeling  it 
at  all. 

Challenge  5:  Real-time  Interaction 

The  ability  of  training  systems  to  provide  relatively 
real-time  interaction  is  an  important  part  of  the  team 
being  able  to  focus  on  team  interaction  and  team  goals 
rather  than  the  shortcomings  of  the  simulation.  A  more 


critical  aspect  might  be  the  real-time  interaction  of  the 
individual  tutors  especially  the  synchronization  of  team 
state  models.  Any  significant  delays  could  result  in  an 
individual  tutor  making  instructional  strategy  decisions 
based  on  inaccurate  information.  Additional  research  is 
needed  to  determine  what  constitutes  a  “significant 
delay.” 

Any  framework  for  a  team  tutor  must  account  for 
optimal  solutions  to:  where  (locally  or  centrally  via  a 
server)  states  are  determined;  how  frequently  states 
need  to  be  reassessed;  and  what  information  is  to  be 
distributed. 

Passive  Sensing  Methods  in  Team  Tutoring 
Environments 

This  section  discusses  available  passive  sensing 
methods  found  in  the  literature  and  how  these  methods 
might  support  or  fall  short  in  supporting  the  concept  of 
a  team  tutor.  Passive  sensing  methods  were  chosen  so 
as  to  minimize  interference  with  the  learning  process. 
Our  goal  was  to  find  accurate,  low-cost,  portable,  real¬ 
time,  passive  sensing  methods  that  would  be 
compatible  with  available  technology  (e.g.,  laptops  or 
other  mobile  computing  platforms). 

Using  Conversation  to  Assess  Affect 

Conversational  patterns,  content  and  flow  have  been 
used  to  assess  affective  state.  Frequent  conversation 
patterns  (D'Mello,  Craig,  Sullins  and  Graesser,  2006; 
D'Mello,  and  Graesser,  2007)  have  been  used  to 
predict  affect  (i.e.  confusion,  eureka,  frustration)  and 
have  been  tied  to  ITS  instructional  strategies  (e.g., 
pumps,  hints  and  assertions)  to  influence  the  trainee"s 
progress. 

While  these  methods  meet  our  goals  for  passive 
sensing,  a  major  drawback  to  these  approaches  relative 
to  our  team  tutor  design  goals  are  the  labor  intensive 
nature  of  the  data  collection  and  analysis  which  limits 
the  tutors  ability  to  provide  a  real-time  assessment  of 
the  trainee"s  affective  state.  Another  limitation  to 
consider  is  that  conversational  data  is  not  always 
present  to  support  affective  assessment. 
Psychophysiological  sensing  methods  offer  a  distinct 
advantage  over  conversational  sensing  methods  in  that 
data  is  always  available  (Parsons,  2011). 

Using  Trainee  Actions  to  Assess  Affect 
Trainee  behaviors  were  used  to  unobtrusively  detect 
mood  and  determine  the  relationship  between  mood, 
performance  and  the  selection  of  successful  coaching 
strategies  (Zimmermann,  Guttormsen,  Danuser  and 
Gomez,  2003).  Passive  sensing  included  capture  of 
control  selection  rates  (Sottilare  and  Proctor,  in  press ) 
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and  mouse  movement  rates  (Sottilare  and  Proctor,  in 
press);  Zimmermann,  et  al,  2003).  Significant 
relationships  exist  between  pleasure  and  dominance 
mood  variables  (Mehrabian,  1996),  trainee 
performance,  a  priori  knowledge,  state  and  action 
variables  (e.g.,  control  selection  and  mouse  movement 
rates)  (Sottilare  and  Proctor,  in  press ;  Wingrave, 
Hoffman,  LaViola,  and  Sottilare,  2011).  These 
approaches  were  affordable,  accurate  and  passive. 
Limitations  of  this  research  were  their  discrete  self- 
report  measurements  of  mood  which  was  only  assessed 
during  pre -training  and  post  training  and  not 
continuously  during  training  to  provide  real-time 
assessment  of  trainee  affect.  Expanded  modeling  of 
affect  using  other  (non-survey)  methods  will  improve 
the  accuracy  and  real-time  support  of  these  techniques 
to  detect  individual  differences. 

Using  Facial  Changes  to  Assess  Affect 
Eye  tracking  has  been  used  in  an  embedded  training 
tutor  (Zachary,  et  al,  1999)  to  assess  what  the  trainee 
viewed,  when,  and  for  how  long,  but  was  not  used  to 
assess  affect.  The  same  or  similar  technology  could  be 
used  to  determine  the  trainee"s  level  of  engagement. 
Given  the  state  of  camera  technology,  a  laptop  camera 
could  be  used  to  measure  changes  in  pupil  dilation. 
Significant  relationships  were  assessed  between  six 
universal  emotions  (sadness,  joy,  anger,  fear,  disgust 
and  surprise)  and  six  facial  measurements  (Dl:  the 
opening  of  the  eye;  D2:  distance  between  the  interior 
corner  of  the  eye  and  the  eyebrow;  D3:  opening  of  the 
mouth  in  width;  D4:  opening  of  the  mouth  in  height; 
D5:  the  distance  between  the  eye  and  eyebrow;  and  D6: 
the  distance  between  the  corner  of  the  mouth  and  the 
external  corner  of  the  eye  -  Neji,  and  Ben  Ammar, 
2007). 

Assessing  Team  Performance 

One  of  the  challenges  of  developing  a  team  training 
system  is  the  challenge  of  assessing  team  performance. 
Typical  team  simulation  trainers  accomplish  this  in  two 
manners,  through  After  Action  Review  (AAR),  and 
through  human-in-the-loop  feedback.  These  traditional 
methods  are  simply  not  an  option  for  a  team-based  ITS, 
which  much  performance  assessment  as  part  of  the 
course.  Research  in  the  area  of  team  performance 
measurement  contains  many  helpful  suggestions  on 
how  to  automate  this  assessment. 

A  recent  review  of  performance  measurement  in 
simulation  recommended  the  following  series  of  ,best 
practices"  in  order  to  automated  team  performance 
measurement  (Salas,  Rosen,  Held,  and  Weissmuller, 
2009).  Performance  measurement  works  best  when 
performance  from  multiple  sources  can  be  captured. 


when  the  assessment  is  tightly  coupled  to  the  action  to 
take,  when  validated  expert  models  perform  the 
assessment,  when  it  directly  supports  learning,  and 
when  it  is  able  to  provide  real-time  corrective 
feedback.  Additionally,  Salas  indicated  that 
performance  assessment  should  be  divided  into  two 
phases:  a  process  phase  and  an  individual  phase.  They 
draw  the  line  between  meeting  a  team  goal,  and  doing  a 
good  job  individually.  ITS  systems,  unlike  their 
human  counterparts,  are  able  to  provide  real-time 
feedback  to  all  of  the  individuals  simultaneously  and 
can  collate  multiple  assessment  sources. 

In  addition  to  the  team  performance  assessment,  each 
individual  must  be  assessed  based  on  contribution  to 
the  joint  goals  of  the  team.  This  is  a  problem  that  has 
been  well  studied  in  the  domains  of  agent-based 
systems  (Yen,  Yin,  loerger,  Miller,  Xu  and  Volz, 
2001),  human-computer  interaction  (Lewis  and  Wang, 
2009),  and  human  performance  (Rothrock,  Cohen,  Yin, 
Thiruvengada  and  Nahum-Shani,  2009).  The 
conclusions  in  these  domains  are  similar  to  the 
conclusions  found  above.  A  solution  to  individual 
assessment  is  presented  by  determining  when  each  user 
is  able  to  take  an  action,  the  amount  of  time  delay  each 
user  has  introduced,  and  the  value  of  the  taken  actions. 
Team  assessment  might  be  performed  in  similar 
manner. 

APPLICATION  SCENARIOS  FOR 
DISTRIBUTED  TEAM  TUTORS 

Two  scenarios  are  put  forth  to  illustrate  how  a 
distributed  ITS  architecture  might  be  used  to  support 
team  training.  The  first  is  a  hypothetical  military 
training  scenario  and  the  second  is  a  hypothetical 
medical  scenario. 

Distributed  Team  Training  for  Military  Search  and 
Rescue  Operations 

Search  and  rescue  missions  are  multi-dimensional 
involving  many  different  military  assets.  For  example, 
in  the  aftermath  of  Hurricane  Katrina,  National  Guard 
assets  (air  and  ground)  conducted  coordinated 
operations  to  rescue  people  stranded  by  the  floods. 
Training  for  these  kinds  of  operations  would  be  nearly 
impossible  on  this  kind  of  scale  in  a  live  (real) 
environment.  The  uses  of  simulations  (e.g.,  games  or 
virtual  worlds)  offer  an  opportunity  to  train  complex 
missions  without  significant  risks  to  the  trainees  who 
are  in  various  geographic  locations.  The  shortfall  in 
these  simulations  is  the  lack  of  a  team  tutor  so  this  task 
is  often  left  to  a  cadre  of  facilitators  who  are  often  not 
co-located  with  the  pilots,  aircrews,  ground  vehicle 
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crews  and  dismounted  soldiers  who  are  performing 
tasks  during  the  training  session. 

The  objective  of  many  large-scale  training  scenarios  is 
to  exercise  the  decision-making  skills  of  the  trainees.  It 
is  desirable  that  the  trainees  experience  some  level  of 
affect  (e.g.,  stress)  and  are  able  to  continue  functioning 
as  a  reliable  team  member.  A  distributed  tutor  with  the 
capability  to  assess  affect  and  manipulate  the 
simulation  to  support  the  individual  and  team  training 
objectives  would  also  be  desirable. 

Distributed  Team  Training  for  Medical  Emergency 
Management 

Municipalities  throughout  the  United  States  conduct 
emergency  management  exercises  based  on  medical 
crisis.  A  team  tutor  would  be  useful  in  providing  a 
distributed  capability  to  understanding  the  performance 
of  team  members. 

Emergency  management  scenarios  include  the 
coordination  of  first  responder  assets  (e.g.,  police  and 
firemen),  hospital  space  and  ambulances.  The  ability  to 
practice  communications  across  the  various  disciplines 
required  to  deal  with  emergency  medical  incidents  is 
critical.  An  understanding  of  the  trainees  and  the 
training  domain  (e.g.,  medical  emergency 
management)  would  enable  a  team  tutor  to  provide 
valuable  decision  support  to  trainees  at  the  junction  of 
critical  decisions.  The  distributed  nature  of  the  tutor 
would  enable  trainees  that  normally  do  not  work  in  the 
same  location  to  understand  the  limitations  of  voice 
and  written  communications  methods. 

CONCLUSIONS  AND  RECOMMENDATIONS 
FOR  FUTURE  RESEARCH 

Significant  technologies  (tools  and  methods)  exist  to 
support  a  distributed  ITS  architecture  for  team  training, 
but  additional  research  is  required  to  fully  realized  the 
lofty  design  goals  set  forth  in  this  article.  Research  to 
mature  distributed  team  tutoring  technologies  should 
continue  with  significant  emphasis  on:  passive 
behavioral  and  physiological  sensing  methods; 
accurate,  real-time  classifiers  to  assess  the  cognitive 
and  affective  states  of  trainees;  and  optimal 
instructional  strategy  selection  techniques  based  on 
limited  trainee  modeling. 

Future  research  should  also  include  enhancements  to 
conversational  pattern  recognition  and  other  sensor 
methods  to  allow  streamlined  computing  methods  to  be 
used  on  laptops  and  other  mobile  computing  devices 
(e.g.,  Blackberry,  Android,  Treo  or  iPhone). 
Additional  research  is  needed  to  make  it  easier  to  setup 


methods  (e.g.,  emote  aloud  sensing)  to  evaluate  affect 
through  prosody  (e.g.,  rhythm,  stress  and  intonation  of 
speech). 

It  will  be  critical  to  be  able  to  measure  team  learning 
and  understand  the  unobtrusive  modeling  of  cognitive 
skills  that  include  remembering,  understanding, 
applying,  analyzing,  evaluating  and  creating  (Anderson 
and  Krathwohl,  2001)  for  collaborative  team  tasks. 

Approaches  that  consider  training  across  a  soldier"s 
career  vice  a  single  training  event,  will  be  enabled  by 
additional  research  in  portable  career  learning 
management  systems.  The  development  and 
maintenance  of  a  trainee  model  specific  to  an 
individual  over  the  course  of  his/her  career  will 
highlight  learning  trends  and  be  useful  to  initialize 
whatever  training  system  the  trainee  might  use. 

Based  on  successes  in  sensor  technology,  researchers 
can  begin  quantifying  significant  relationships  between 
behaviors/physiological  measures  and  trainee  states, 
and  then  trainee  states  and  successful  instructional 
strategies.  The  key  is  to  evolve  a  more  comprehensive 
trainee  model  without  the  use  of  self-report  methods 
that  can  interfere  with  the  training  process. 
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